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ABSTRACT 


An algorithm Is presented for using a robot system with a single 
camera to position In three-dimensional space a slender object for 
Insertion Into a hole; for example, an electrical p1n-:ype termination 
Into a connector hole. The algorithm relies on a control -configured end 
effector to achieve the required horizontal translations and rotational 
motion, and It does not require camera calibration. A force sensor in 
each fingertip is integrated with the vision system to allow tne robot to 
teach itself new reference points when different connectors and pins are 
used. Variability in the grasped orientation and position of the pin can 
be accomodated with the sensor system. Performance tests show that the 
system is feasible. More work is needed to determine more precisely the 
effects of lighting levels and lighting direction. 
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1.0 INTRODUCTION 


Control engineers frequently are faced with the problem of 
controlling a device to perform a task for which It is not particularly 
well -suited. A simple example of this Is the two-link robot arm shown in 
Figure 1(a). In order to obtain straight-line motion of the endpoint in 
the y-direction, the motors at the two joints must properly control the 
angles 0 ^ and This is a nonlinear problem both kinematically 
and dynamically [1, Chapter 6], [2, Section 9.1]. Also, the coupling 
between the links means that both motors must perform accurately even 
though straight-line motion is desired in only one direction. Of course, 
there are reasons for using such a two-link arrangement, for example, to 
achieve a large workspace. 

However, suppose that we require straight-line motion only over a 
relatively short distance In the workspace. An easily-controlled device 
for doing this Is known as an xy-translatlon table, and Is shown In 
Figure 1(b). It consists of two perpendicular lead screws with motors. 
One screw unit Is mounted on a plate, which Is In turn mounted to the 
second screw unit. Suppose we attach the plate to the endpoint of the 
two-link arm, and the other screw's nut to a gripper. Then we could 
easily obtain straight-line motion for the gripper by driving the motors 
In a fixed speed ratio. Note also that If pure translation Is desired In 
either the y or the x direction, then the motion error Is due to only one 
of the motors, and Is not cumulative. 

Devices or systems like the one In Figure 1(b) that are designed from 
the start with ease of control In mind are said to be 
control - conf 1 gured . As modern systems become more complex, their control 
becomes more difficult, and It Is Important that they be 
control -configured early In the design process. This Is especially true 
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(b) 

Figuro 1. Achieving Straight-Line Motion. 

(a.) The two-link manipulator has 
nonlinear kinematics, (b) The 
platform-lead screw mechanism has 
linear kinematics. 




In robotic systems, where complex kinematic relationships are quite 
common. In addition to choosing kinematic arrangements that are easily 
controlled, a system can also be control -configured by proper sensor 
Integration; that Is, proper choice of sensor locations and sensor types 
to complement each other's capabilities and to satisfy the needs of the 
control algorithm. 

In this paper we discuss a visual servolng algorithm and the design 
of a control -configured robotic end effector to Implement that 
algorithm. The algorithm uses a single camera to guide the end effector 
in positioning the workpiece. Only v.wo-dlmenslonal Information Is 
available from a single stationary camera. However, with the proper 
control strategy and sensor integration, positioning In the 
three-dimensional space can be accomplished. The control algorithm to be 
presented integrates the camera Information with feedback from a force 
sensor on the end effector to position a slender object for Insertion 
Into a hole. 

There has been much published on the "peg-1 n-hole" Insertion 
problem. This can be a difficult problem where tolerances are tight, and 
the success rate can be improved by the use of a remote-center compliance 
(RCC) device. However, In our application, the Insertion process Is not 
difficult because the tolerances are large and because of the Inherent 
compliance In the fingertips of the end effector. In this paper we are 
concerned primarily with the positioning process necessary to align the 
workpiece above the hole prior to Insertion. 

Few general methods are available for the design of 
control -configured end effectors. Palm [3] discusses the control of a 
non-anthropomorphlc hand configuration for manipulating cylindrical 
objects In five degrees of freedom. The hand Is easy to control because 
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the motion of each degree o* freedom can be produced without requiring 
complicated coordination of more than one motor. Some kinematic and 
control principles useful for the design of control -configured hands are 
given by Datserls [4], 

2.0 CONTROL FOR ASSEMBLY APPLICATIONS 

Assembly operations for products that change slightly from one model 
to another cannot be performed utilizing fixed automation techniques. 
One such problem Is the assembly of electronic wire harnesses. Small 
quantities of each type of wire harness are assembled at oie time, and 
each wire harness may consist of different types of components and 
wires. The development of a flexible system for the Insertion of 
electrical pins Into a connector would offer a partial solution to 
automating the assembly. Although motivated by the win? harness 
application, the results to be presented are generic enough to be 
applicable to the general problem of using a single camera to position an 
object. 

A progranmable robot Is a vital part of the assembly operation due to 
the flexibility required by the operation. A robot like the PUMA series 
560/600 has good repeatability but Is less accurate when trying to 
position at a calculated point. Accuracy Is then only achieved by 
teaching the point to the robot. This operation decreases the 

flexibility of the system unless sensors can be used to allow the robot 
to teach Itself the required reference points as new situations arise. 
The results presented here make use of vision and force sensors to give 
the robot this capability. 

We have used the following wire harness assembly scenario In 
developing the results. In order to avoid the need for manual setup, one 
or more robots place the connectors In fixtures and also route the 


wires. The fixtures hold the connectors firmly throughout the 
operation. Thus the robot knows the general location of each connector 
but there may be some uncertainty In the rotational orientation of the 
connector and In the height of the connector's surface*. By means of a 
registration mark on the connector It Is possible to use machine vision 
techniques to approximately locate the ordered holes In the connector 
[5]. But since these are calculated points the robot will have some 
positional Inaccuracies unless It can automatically teach Itself a 
reference point on the connector. Once this has been done the robot can 
begin Inserting wires with pin-type terminations Into the proper holes. 
In this paper we focus on thr. algorithms required to teach the reference 
point and to visually servo the pin to the Imprecisely-known hole 
location. 

After several wires have been Inserted into the connector. It becomes 
Impossible to use any visual servoing technique to find the holes. 
However, once the first two holes are located by the method to be 
presented, the angular orientation of the connector In the horizontal 
plane Is known. Therefore the locations of the holes can be supplied to 
the robot from the geometric data base for the connector. Because the 
holes are relatively close together, there will be negligible error In 
moving to them even though they are calculated points. 

There have been other Investigations of the use of visual servoing 
for assembly applications. For example, Agin [6] and Berger [7] discuss 
geometries similar to the one to be presented here, but they use a camera 
attached to the end effector. This arrangement presents other 

difficulties. Others are Investigating the use of two cameras for 
stereoscopic vision. But little Is known about the effective use of a 
single camera Integrated with other sensors to produce a system capable 
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of automatically teaching Itself new reference points, and of using this 
capability to perform Insertion operations In an adaptable manner. 

3.0 SYSTEM CONFIGURATION 

The system used to test the concepts Is shown In Figure 2. The 
system elements were selected partly because of availability, and do not 
necessarily constitute our recommendations for a commercial 
Implementation. For example, the algorithm can be run on a much smaller 
computer than a VAX 11/780. The PUMA robot, the manipulator and the 
vision system are all controlled and/or monitored by e microcomputer (a 
DEC DCT11-EM). The PUMA robot contains Its own controller (LSI-11/2) and 
system software (VAL). The PUMA also has a digital I/O module that can 
be used to conmunlcate with external devices. It Is the digital I/O 
lines that the DCT11-EM uses to see If the robot has completed a motion, 
to signal the robot to proceed to next step, or to terminate the present 
move. The I/O lines are triggered or monitored through a parallel port 
on a data acquisition Interface. 

The end effector was designed specifically to aid the Implementation 
of the visual servolng algorithm to be discussed. The end effector 
consists of an xy positioning wrist and a set of fingers whose tips 
contain a platform capable of rotating a part about an axis that Is the 
common normal to the fingers. This rotational capability assists In 
acquiring the pin because acquisition Is easier when the pin's long axis 
Is perpendicular to the long axis of the fingers (see Figure 3). The 
fingers have force sensing capability along their long axis (the 
direction of Insertion). A more detailed description of the end effector 
Is given In Section 5.0. 


6 



Figure 2.. System Configuration 
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Figure 3. Plr. Acquisition with a Rotating Platform 
on the Flngertipa. (a) The wires do not 
Interfere when the pins are fed horizontally, 
(b) The pin must be rotated to the vertical 
for the servolng process. 
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A Hitachi KP-120 CCTY canrra Interfaced to a VAX 11/780 Is utilized 
for vision feedback to control the manipulating end effector. The 
Hitachi camera consists of a solid state pickup device with a resolution 
of 320 by 244 pixels. All communications between the vision routine In 
the VAX and the DCT11-EM are performed through the RS-232 port. 

The connector where the Insertions are to be performed Is mounted 
Into a simple fixture that holds It rigidly and vertically so that the 
top face of the connector Is horizontal. The face of the corrector used 
In our tests Is 25nm In diameter aid has thirteen holes 0.8 nvn In 
diameter. However, our methods should be applicable to any connector 
that has holes for pin-type terminations. 

Lighting conditions are provided by two lights, one to process the 
holes on the connector, the other to highlight the pin. The lighting 
setup Is shown In Figure 4. The back light Is used only when the 
connector Is being analyzed for the hole locations. The front light Is 
on at all times and reflects off the pin when being positioned. When the 
pin Is being positioned the back light Is switched off. 


4.0 CONTROL ALGORITHMS 

Visual servotng Is utilized to position the robot In the vertical 
direction* (y w ) and also to control the manipulating wrist In the 
horizontal directions (x w and z w ). 

4.1 Geometric Analysis 

The target hole In the connector and the pin must both be In the 
Image field of view, and this can be accomplished by mounting the camera 
off to one side of the connector stand and at an angle to the 
horizontal. Figure 4 shows the oblique mounting of the camera. Only two 
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Figure 4. Lighting Arrangement 


ORIGINAL PAflS Ifr 
OF POOR QOKLTCfi 



dimensional Information can be extracted from a camera Image. No depth 
Information along the axis perpendicular to the camera Image can be 
calculated. For a fixed camera placed at an angle to the surface the 
height of the pin over the hole in the Image Is related to the actual 
height of the image from the hole, the angle of the camera, and the 
magnification of the lens. 

The acqusltion of an Image utilizing a camera can be mathematically 
modelled by replacing the lens with a pin hole. The pin hole camera Is 
the simplest of all cameras. In theory only one ray from a point In 
space can pass through the Infinitesimally small hole at the center where 
the lens is supposed to be. The xyz coordinate system used is located at 

the lens center, and is shown in Figure 5 (the x coordinate points into 

the page). Let the distance q be the fixed distance corresponding to the 
distance between the lens center and the Image plane. The focal length Is 

f. The coordinates of a point in the Image plane are x { and y^ (see 

Figure 5). 

From similar triangles a point on an object at location (x,y,z) Is 
related to the image coordinates x.j, y^ for a pin hole camera as 


-qx/z 

(1) 

-qy/z 

(2) 


The above equations show that for a pin hole camera the position of the 
point In the Image Is related to the object's perpendicular distance away 
from the principal ray and the distance along the principal ray. For a 
camera placed at an angle from 1 * 1 * horizontal a motion In the horizontal 
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Figure 5- The Geometric Model 
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plane causes both a change In the distance of the object from the 
principal ray and a change in the distance along the principal ray. 

For the experimental setup a lens with a 75 irm focal distance is used 
with a 10 mm extender. This makes the total required distance of the 
image from the lens 85 mu. From the lens equation 1/q+l/p * 1/f the 
distance p from the object to the lens is calculated to be 637.5 mm. 
This Is the distance of the position of the connector from the lens along 
the principal ray of the camera (see Figure 5). The maximum horiontal 
distance of travel of a pin across the connector Is 25mm, and the camera 
is mounted at about 30.0 degrees from horizontal (Q). For this geometry 
we can use a linearization of Equation (1) about the value z * p. Thus 
the change In the Image coordinate y^ Is approximately given by 

dy^ = -q dy/p (3) 

The linearization of Equation (1) has the effect of eliminating the 
perspective transformation that normally will occur when using the pin 
hole camera model. With this linearization our camera model Is described 
by the geometry of Figure 5. When the perspective transformation Is not 
considered, the complexity of the vision analysis Is greatly reduced. 
The following sections describe how a pin can be servoed using the 
linearized analysis so that It can be positioned above a hole. 

4.2 Horizontal Positioning 

Since q and p are constants, the change In y^ is linearly dependent 
on the change In the object's position In the direction perpendicular to 
the principal axis (y). If two locations, numbered 1 and 2 In Figure 5 
are a distance D away from each other the change In position of the two 
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holes In the Image will be given by dy^. This Is the same change that 
occurs for the motion of the pin a distance D. Therefore the height of 
the pin image above the hole Image for position 1 Is equal to the height 
of the pin Image above the hole Image for position 2. Therefore we 
position the pin above a reference point on the connector and move the 
pin horizontally (in the negative z y# direction), thus maintaining a 
constant pin height above the connector surface. 

By keeping the height of the pin above the connector constant It Is 
possible to calculate how far above the hole image the tip of the pin 
image must be so that the pin Is aligned above the hole. From Figure 5: 
dy a distance between pin and hole along the ax's 
perpendicular to the principal ray 
m = magnification factor of the lens 

s q/p 

H * height of the pin above the hole In world 
coordinates (x w , y w , z^) 
h * height of the pin above the hole In image 
coordinates (x^ # y^ ) 

We wish to calculate h. From equation (3) 

h ■ -q dy/p = -m dy 
From trigonometric relationships 


dy * H cos (Q) 
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Therefore 


h = -mH cos (Q) (4) 

Variables dy^ and h, from equations (3) and (4) repectlvely, are 
along the y-axl s of the camera. The x-axIs of the camera Is aligned 
parallel to the world \-axis and any motion In the world x^axIs 
does not constitute a change In the position along the principal ray* 
Therefore from equation (1), x^*-qx/p, and a change In the x direction 
(dx) causes a change In the camera coordinates of dxj*-(q/p) dx. Thus 
the inherent nonlinear problems that occur when investigating the change 
in the y-axis of the Image due to a positional change D are not present 
In the x direction. 

Since calculations are done In pixel size and not In actual units of 
distance, a simple approach Is to calibrate the camera by actually 
placing the pin directly above the hole at an arbitrary height and noting 
on the image what the difference is between the pin position and the hole 
center In pixels. When calibrated with this method the actual height H 
of the pin and the camera angle Q are not necessary. 

4.3 Accomodating for Variable Pin Height 

As discussed previously, one Important factor for the Insertion of 
pins Into a connector for the assembly of wire harnesses Is flexibility, 
since different types of connectors and matching pins have to be 
considered. Calibrating the camera for each assembly operation In the 
manner described above can be a very time consuming operation. Also, 
differing pin lengths and variability In grasp location mean that the pin 
tip will not always be at the same height above the connector surface 


unless the control software can compensate for this variability. Figure 6 
shows the error that can occur If the pin Is not placed at the correct 
height above the hole. 

The problem of variable pin height can be solved by placing the pin 
directly above a fixed reference point on the connector. This point's 

Image location has been previously recorded. The pin Is then servoed 
down towards the reference point until the tip of the pin In the Image Is 
at a predetermined pixel height above the Image location of the reference 
point. This same pixel height Is then utilized to position the pin above 
the desired hole for Insertion. The pin Is servoed towards the camera 
until It appears In the Image above the hole at the same pixel height 
that was used with the reference point. 

4.4 Establishing the Reference Point 

To make the assembly operation less dependent on an operator It Is 
possible to have the robot choose the reference point. It Is assumed 
that the robot will have a general Idea where the connector Is located 
since In the overall wire harness assembly the connector could be placed 
In the vise by the robot. Initially the robot grasps a special pin with 
a small LED mounted on the end and facing the camera. This pin Is wider 
than the holes In the connector so that It can not be Inserted Into any 
of the holes. The pin Is lowered down towards the connector until the 
pin touches the connector surface (this Is detected by the force sensor 
in the fingertips). At this point an Image is taken of the LED and 
recorded. In this way the reference point can be positioned at the 
connector surface height, and this height Is then known to the robot. In 
the manner described above, the pin is positioned above the reference 
point and servoed downwards until It appears at a predetermined pixel 
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Figure 6 . Positioning Error Due to 
Incorrect Pin Height 
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height above the reference point. Figure 7a shows the procedure as seen 

along the x axis. Figure 7b shows the view from the camera. 

The visual analysis uses small window strips at two positions. These 

windows are 40 pixels longer than the connector window and four pixels 

high. Window strip 1 Is placed 30 pixels above the recorded Image 

location where the LED pin touched the connector surface. Window strip 2 

Is placed 30 pixels above the Image location of the connector hole. 

These strips are shown In Figures 5 and 7. Initially, the pin tip Is 

• 

placed high enough so that It will not appear In either window. The pin 
Is then moved down In the negative y w direction until the pin enters 
window 1. Then the pin Is moved forward In the negative z w direction 
until It enters window 2. The pin then has the same coordinate as 
the connector hole. At this point the ' pin Is moved In the x w 
direction, which coincides with the x direction. 

For servolng along the x w -ax1s it Is assumed that the pin has 
already entered the Image strip. The center of gravity of the pin In the 
x Image direction Is calculated and compared to the x coordinate of the 
hole Image. When the two align the vision routine signals the micro- 
processor to stop the motors. The same routines that are utilized to 
find the center of gravity and bottom of the LED are used to locate the 
the pin tip and to see If there Is a pin actually In the Image. 

4.5 Summary of the Procedure 

The following steps constitute the procedure for the assumed scenario. 
1. The robot places a connector In the fixture. 
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2. The robot acquires the LED pin and touches the surface of the 
connector. The Image of this point Is recorded. The connector 
height Is now known In terms of a pixel location. This Is reference 
point 1. Window strip 1 Is placed 30 pixels above this point. 

3. The robot replaces the LED pin and begins the Insertion process. It 
acquires a new pin but does not know exactly how far this pin 
protrudes from the fingertips. 

4. The end effector Is moved to a point directly above reference point 
1, at an end effector height high enough to allow the tip of the 
longest pin to be above window strip 1. 

5. The robot then servos down until the pin tip enters window strip 1. 
The pin tip Is now at a known height directly above reference point 1. 

6. The robot servos the pin forward (toward the camera) until the pin 
tip enters window strip 2. The pin tip now has the same z w 
coordinate as the connector hole. 

7. The end effector servos the pin sideways (In the x direction) 

FT 

until the pin Image lines up vertically with the hole Image. The pin 
Is now directly over the hole. 

8. The robot servos the pin downward until the Insertion detent force Is 
detected by the force sensor. 

5.0 DESIGN OF THE END EFFECTOR 

From the above procedure we see that the following motions are 
required of the system: 

1. Robot motions: 

a) acquisition of the pins and transportation to the general 
vicinity above the connector; 


b) vertical motion (along the y w direction) 

2. End effector motions: 

a) fingertip grasp for holding the pin; 

b) platform rotation at the fingertips, for pin acquisition 
and vertical alignment (see Figure 3); 

c) horizontal translation In the z w direction; 

d) horizontal translation In the direction. 

The end effector designed to Implement these motions In an 
easlly-controllable manner Is shown In Figure 8. It has a wrist that 
consists of two sections. Each section has a motor-driven lead screw 
that provides a translation for whatever Is connected below It. When the 
wrist Is horizontal and facing the camera as shown In the lower left-hand 
drawing of Figure 8, the lead screws provide translations of the pin In 
the z w and x w directions, as required for the servolng algorithm. 

The opening and closing of the fingers Is pneumatically actuated In 
an on-off manner. The rotation of the fingertip platforms Is produced 
with a motor driving a flexible shaft. Each fingertip contains a 
strain-gage sensor for measuring forces In the y w direction. 

6.0 DETERMINATION OF PIN ORIENTATION AT ACQUISITION 

In addition to simplifying the pin acquisition process as shown In 
Figure 3, the rotating platforms on the fingertips can be used to 
determine the location and orientation of the pin when It Is acquired. 
Figure 9 shows an arbitrary situation at acquisition. Two pairs of 
Infrared through -beam sensors are mounted on the fingers at the rim of 
the rotating platform. The emitters are on one finger; the detectors on 
the other. With two pairs of such sensors, we can determine the pin 
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Three View* of the End Effector 





orientation by rotating the pin with the platform no more than 180°. 
The 180° limitation Is due to the trailing wire on the back of the pin. 

When the pin Is acquired the platform Is rotated counterclockwise. 
The stepper motor controller monitors the angular rotation needed to 
interrupt sensors 1 and 2; these angles are denoted ^ and ^ 2 . 
From trigonometry It can be shown that the pin's angular orientation 9 
and offset c are given by • 


tan 9 - (C, ♦ 02)7(3! ♦ S 2 > (5) 

c » aR(S ] - s 2 )/b (6) 

where 

* sin $ ! (7) 

Cj -cos & ! (8) 

a - ^ 2/(HC 1 C 2 - S^) (9) 

b * 2 (S 1 s 2 - Cfy) (10) 

R ■ platform radius (11) 


The orientation 9 is used to align the pin vertically prior to the 
servolng process showi. In Figure 7. Because the center of the platform 
Is Initially aligned above reference point 1, the offset c would cause 
the pin to miss the hole by the distance c. But since c can be 
calculated from (6), we can compensate for Its effect when the wrist 
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7.0 PERFORMANCE TESTING 


This section describes the results of testing the control algorithm. 
The tests were performed at a pin height above the hole that corresponded 
to approximately 3mm above the connector. A series of pin positioning 
trial runs showed that the pin could be positioned above the desired 
hole. It was discovered that even If the pins were slightly off vertical 
there was enough compliance In the fingers to allow for Insertion. 
However, the method Is somewhat sensitive to lighting conditions. 

The main use of the front light Is to brighten the pin. The ideal 
lighting configuration is to have the light positioned so that the 
majority of the light Is reflected off of the pin and Into the camera. 
Unfortunately, If the light is positioned below the connector to achieve 
total reflection into the camera. If the pin moves near the back of the 
connector the light Is blocked by the connector Itself. 

The optimum configuration Is to have the light tilted about 10° 
above horizontal. If the pin Is not rotated but kept vertical, the light 
that Is reflected Into the camera Is diffused lighting from the pin, 
which lowers the Intensity of the pin In the Image. If not enough light 
Is reflected off the pin Into the camera, the pin will blend Into the 
background. 

The type of pin that was used for the assembly was a silver color pin 
about 25 ran long. With this type of pin It was possible to obtain enough 
contrast. No detailed study was done on the reflectivity of different 
types of pins but In general gold pins do not tend to be as bright as the 
silver pin that was used In the experiment. 


8.0 CONCLUSIONS 


The performance characteristics that were desired were: 

1. Accurate positioning of the pin over the desired hole for 
insertion. 

2. Flexibility In the system so that different pins and 
uncertainties In grasp location can be easily accomodated. 

The performance tests show that the robotic assembly of pins Into a 
connector In a flexible but accurate manner is feasible. It was shown 
that with the use of visual feedback from one camera a pin of any length 
can be acquired and positioned accurately enough over the desired hole 
for insertion. With the combination of force feedback the procedure can 
be automated such that the robot Itself can locate a reference point on 
the connector. This eliminates the need for an operator to teach the 
robot a reference point every time a different assembly Is started. 

It was shown that, by using a vision algorithm that relies upon the 
relative distance between where the pin Is and where It should be, costly 
set up times could be eliminated since the camera need not be calibrated 
to the robot frame. The elimination of camera calibration Is very 
Important, for an operation where changes are to be made often In the 
assembly operation. 

The success of the control algorithm Is due partly to the design of 
an end effector that easily gives the motions required by the control 
algorithm; namely, accurate horizontal translation In two directions 
across the connectors' surface, and rotation at the fingertips to ease 
pin acquisition and to optimize light reflection off the pin. 
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